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A test based on the Operations Analysis Curriculum 
at the United States Naval Postgraduate School was 
administered to 104 Naval Officers. All examinees were 
graduates or students of the Operations Analysis Curricu- 
lum and/or officers holding Operations Anslysts billets in 
the Navy. The sub-sample, 34 examinees, consisting of 
officers holding Operations Analysts billets and/or Operations 
Analysis graduates was not sufficient to make adequate 
statistical determination of the measure of effectiveness 
proposed in a suggested methodology. The data gathered 
did crudely support hypothesized learning and forgetting 
curves and suggested that the effectiveness of Operations 
Analysis graduates assigned directly to Operations Analysts 
billets immediately after graduation is much enhanced compared 
to graduates who are returned first to fleet operational 
billets. The effectiveness of Operations Analysis trained 
officers in Operational Analyst billets was shown to be 
quantitatively and subjectively significantly superior to those 
with no formal Operations Analysis training. These results 
indicate that Naval assignment policies should be reviewed 
in hopes of assigning more Operations Analysis trained 
officers (consistent with other requirements) to these 
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billets. Further investigation of the results of the test 
vehicle and other statistics common to Operations Analysis 
graduates yielded a feasible procedure with which to 
augment the screening of prospective Operations Analysis 
students. Final Quality Point Rating, an acceptable 
measure of performance, had a .6], correlation with four 
readily available statistics. 

This evaluation suggests that further study in this 
area has great promise in yielding useful measures of 
effectiveness for all personnel filling billets requiring post- 
graduate education, provided a more effective method is 
employed to insure completion of the required test 


instrument(s). 
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In May of 1964 Commander P. D. Roman, and 
Lieutenant Commanders K. D. Russell and J. M. Dunlop 
posed a highly interesting and promising methodology designed 
to evaluate the effectiveness of the utilization of technically 
trained personnel within the Navy. [1] The personnel 
involved in this suggested Methodology were Naval Officers 
who have received postgraduate education. The main 
motivation for this proposed model was the fact that no 
previous quantitative study had been made to determine if a 
better balance of career duty assignments should be devised 
which would allow officers to attain the operational knowledge 
and experience requisite to Military Command and simultane- 
ously approach maximum effective use of their technical skills 
(sub-specialties). Because the acquisition of technical skills 
is costly, as well as necessary, finding an optimal procedure 
to utilize these skills, consistent with other requisites, is 
virtually a "must." The pure rationality of cost-effectiveness 
alone supports this supposition. 

Commander Roman, et. al., constructed a test instru- 
ment modeled on the Operations Analysis curriculum at USNPGS 
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which would, hopefully, measure the native ability and 
technical knowledge of Naval Officers who are holding 
Operations Analysis billets and/or graduates of the Oper- 
ae Analysis Curriculum. It was their hypothesis that 
by categorizing these officers into several distinct groups 
the statistical results of their performance on this test 
would lead to a more efficient system of career planning 
for these officers. If this procedure proved successful 
for the Operations Analysis sub-specialty, it would obviously 
have applications for all officers who are technically trained. 
The authors of the referenced methodology have presented 
an admirable treatise on the need for the execution of such 
a quantitative study. 

In September of 1964 the authors of this paper mimeo- 
graphed the test instrument and distributed it to officers 
holding Operations Analysis "P" coded billets and/or graduates 
of the Operations Analysis Curriculum at USNPGS. A 
total of 34 returns from.a population of 197 officers was 
realized. In January of 1965 the test was also administered 
to 36 second year Operations Analysis students at USNPGS 
scheduled to graduate in May 1965. In November of 1964, 

34 first year Operations Analysis students completed the 
aptitude portion of the test vehicle. 
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The purpose of this paper is three-fold: 

I. To determine the adequacy of the test as a 
measuring instrument, i.e., does it measure what it is 
supposed to measure? 

II. To carry out, where possible, the statistical 
procedures suggested by Commander Roman, et. al., and 
analyse their usefulness and/or implications. 

Ill. To analyse the results of a multiple linear 
regression analysis performed on the CDC 1604 Computer 
to develop a statistical means to augment selection 
procedures for input to the Operations Anslysis Curriculum. 

The following chapter details the analysis of the data 


for each of these purposes. 





CHAR E Rei 


LTHE ANAL YStrsS 


I. Determination of the adequacy of the test instrument. 

The test instrument used in this study consists of a 
background questionaire and three test parts. The question- 
aire is designed to obtain the examinee's educational background, 
a history of his duty assignments, a listing of graduate and 
undergraduate courses completed and the examinees opinions! 
as to what courses he is lacking that are required in his 
billet. The main purpose of this part of the test is to 
stratify the examinee according to educational background. 

The other information gained will provide further refinement 
of this stratification. The adequacy of this questionaire is 
purely subjective. The authors feel that the information 
provided by the questionaire will allow assignment of testees 
to logical categories of interest to this study. 

Part I of this test is an aptitude test designed to 
provide a measure of the examinee's ability in problem solving, 
the only major factor accepted as effecting current measures 
of intelligence. ~ This test was constructed, from investigations 

Ser earaie Federal Education, You're in the Classroom 
Now. Time, 83, 3, January 17, 1964: 72 ~.- 


4 





of aptitudes of high level personnel, under the direction of 
Dr. J. FP. Guilford at the University of Southern California, 
Los Angeles. [1,6,9] The authors will assume that the 
results of this exhaustive and authoritative study has led to 
the formulation of an excellent test to measure native ability. 
It follows that the results of this test will allow examinees 
to be rated according to inherent abilities. This stratification, 
coupled with background areas, will provide an excellent means 
for comparison of technical skills in the selected categories. 
Part II of the test deals strictly with the retention 
of fundamental concepts of the basic courses within the 
Operations Analysis Curriculum: namely, Advanced Calculus, 
Linear Algebra and Probability theory. This test was 
formulated from suggestions of Professors of the Mathematics, 
Physics and Operations Analysis Departments of USNPGS. 
It is indeed a moot question as to whether or not a good 
working knowledge of these courses measures, with any degree 
of accuracy, the ability or success of an individual as an 
Operations Analyst. There can be little doubt, however, 
that proficiency in these fields does reflect some measure of 
the examinees technical abilities. Beacuse we are once again 
caught in a subjective (or qualitative) area, at this point we 
will assume that the combined opinions of these recognized 
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educators provides a good cross section of the technical 
knowledge required of an effective Operations Analyst, and 
the results of Part II of this test will yield, at least, a 
relative measure among the groups of officers of their 
abilities as Operations Analysts. Relative performance is, 
after all, a very important aspect in our real world and 
this relative performance is in essence one major objective 
of this study. 

Part III is a practical test designed to evaluate the 
examinees! ability to recognize the applicability of a class of 
methods to specific problems. The situations presented in 
these problems do not have clean-cut answers, but are 
designed to determine how familiar an examinee is with an 
"accepted or proven" Methodology as related to well-known 
Operations Analysis problems. The answers to the situations 
posed are nothing more than a mean combination of opinions 
expressed by noted analysts and professors at USNPGS. 
Once again we are using the general reasoning of the 
preceding paragraph in stating that this portion of the test 
will display a relative measure of how familiar an examinee 
is with the "accepted" tactics of Operations Analysis. 

Thus far we have been concerned with the question of 
whether the test will measure what we want it to measure. 
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Is it a valid indication of the technical abilities we are 
trying to measure? ‘To this point we have tried to answer 
these questions in the affirmative by the use of subjective 
reasoning. ‘This procedure is necessary because the deter- 
mination of test validity does not readily or easily lend 
itself to meaningful quantitative albasticins Despite this 
dilemma, certain mathematical techniques do allow a degree 
of quantitative determination of adequacy to be calculated. 
The entire question of determining the adequacy of this test 
could be measured by two broad criteria; VALIDITY and 
RELIABILITY. fFeliability is defined as a measure of how 
faithfully the test allows the examinee to display the true 
percentage of the questions presented in the test to which 
he actually knows the answers. A mathematical presentation 
of this reliability follows. 

Doctor J. FP. Guilford has developed a mathematical 
model to evaluate the reliability of any test. Because the 
entire development is extremely lengthy we shall present 
only the basic assumptions and final forms of the reliability 
Panecions.~ 

= eaaligorde J. P. Psychometric Methods, McGraw Hill 
Co., New York, 1954: 36 
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The effects of these variations have been reduced as 
much as possible by the use of an a priori weighting of raw 


total scores. 


_R(K-1) - W 


(1) a 
where: S = Adjusted Score, 
R = Number of Right Answers, 
W = Number of Wrong Answers, 
K = Number of Alternate Responses to each 


item. 
This method of scoring is designed to nullify the small but 
not necessarily minute possibility that the examinee may 
guess the correct answer. 

Doctor Guilford has postulated that, in theory, a 
regression equation could be calculated between observed 
scores and the true score (exactly how many questions the 
student knows the answers to) as depicted in Figure i. 

In the development of this theory Dr. Guilford postulated 

that in any academic area a test of infinite length would be 
necessary to completely describe an examinee's knowledge of 
the subject. In like manner the authors have denoted true 


score by the symbol QQ). 


Oy, 1 Regression line. 

Lit 

> til Kx meee 

ae af Standard deviation of the 
NS. error in the measurement. 
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True Score = © 
FPrevuRE 1 
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The following assumptions were used in the mathematical 


development: 
ai Me = Mean error -al, 
II foe = Correlation of true and error scores = 0. 


Til le, Cp = Error score correlation in any two forms 
of the same test = 0. 


IV Distribution of errors is normal. 
It then follows that: 
Mo = My + Me = My 
and 
Vo = Vy + vey where > = variance of l, 
This leads to the logical definition of reliability of a test 


as the proportion of true variance in obtained test scores 


yo 
Yoo 
— (2) a eats 
Nieie 
where: foe Mest reliability, OoLke ay 
veg = True variance of test scores 
fae = Observed variance of test scores. 


2 
Because of the difficulty in obtaining Poo equation (2) has 


been reduced (closely approximated) as follows: 


y wy" 
_ ~~ = 3 J 
vq" 


(3) et 


om 
| 


where: = Difference between the scores on even 
me items and odd items on the test, 
Veo = Variance of these differences, 
wae = Variance of obtained total scores. 


C 


The variables in equation (3) can be readily calculated for 


any test. 





| A final consideration must be made in view of the 


fact that the test in question was timed and therefore 


speeded to some extent. 


Speed does detract from 


performance which we do not want reflected in reliability. 


The final form of this reliability equation is 


(4) im 
where: kn 
het 
wt 
2 


pil 


he - a 


Reliability of slightly speeded test, 


as before, 


mean number of unattempted items, 


variance of total test error scores. 


Equation (4) is a close approximation to true reliability, as 


previously defined, 


== OAS 


alc 


and 


< |.3, 


la 


As suggested by Commander Roman, 


were placed in the following categories: 


of a speeded test provided that: 


where: NIV 


Vw 


= 
a 
a 


Standard deviation 
of unattempted test 
items, 


Standard deviation of 
test items answered 


incorrectly. 


et. al., examinees 


(1) USNPGS Students nearing completion of their 


final year of the curriculum. 


(1A) USNPGS Students in second term of first year 


of Operations Analysis Curriculum. 
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(2A) Graduates who have been assigned (and are in) 
directly to Operations Analysis billets. 

(2B) Graduates who have never been associated with 
Operations Analysis billets. 

(2C) Graduates who have completed a direct assignment 
in Operations Analysis billets and are now in unassociated 
activities. 

(2D) Graduates who were not immediately assigned 
Operations Analyst billets but are now serving in that 
capacity. 

(3) Non-Graduates who are presently serving in 
Operations Analyst billets. 

Clearly all examinees involved in this study fall into one 
and only one of these categories. The chosen method of 
attack is to compare various categories by means of forgetting, 
learning and re-learning curves in hopes of deducing a useful 
measure of effectiveness. As a common basic for comparison, 
all examinees were stratified by comparing their results of 
Part I (Inherent Abilities) with the mean score of Part I 
for Category I, This procedure is logical because present 
students should be more familiar with the basic fundamentals 


of the courses test Parts II and III are concerned with. 
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To be specific, a ratio of an individual's score on Part I 
to the mean score of Category I on Part I was multiplied 
by the scores on Part II and III and this figure compared 
to the same mean score attained on Parts II and III by 
Category I. 

From equations (3) and (4) and the information displayed 
in Appendix II the reliability of this entire test was calcul- 
ated to be h = .902),. Category I students were chosen as 
a reliability base for the following reasons: 

Il. All data and comparisons are to be based on their 
performance. 

2. Strict time limitations were imposed on these 
testees and we feel that their test conditions are more in 
keeping with the stipulations and assumptions made in the 
development of the reliability equations. 

In the final analysis, validity has to do with what test 
scores measure and what they will predict. A score is valid 
for predicting anything with which it correlates, where 
"anything" does not include the score itself, for a self 
prediction has to do with release We chose to predict 

*emirord: J. P. Psychometric Methods, McGraw Hill 
Co., Inc., New York, 1954: 366 

-Tpid: 398 
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final graduate level QUALITY POINT RATING* of 
Operations Analysis graduates of USNPGS. Here we 
have assumed that the degree of successful completion of 
the curriculum (hence a measure of technical ability) is 
measured by the reliability of the criterion of Q.FP.R.}3 
although the reliability of Q.P.R. as a criterion is not 
known, it is the only quantitative measure we have for 
comparison and have therefore assumed, for the purposed 
of this study, it to be 1.0. The results of a multiple 
linear regression analysis (Appendix IV, pp.50 ) shows that 
the correlation coefficient of Q.FP.R. to various parts of 
the test scores is a minimum of .5978 and a maximum of 
-6142. As shown in Appendix IV, it was found that Part 
II versus Time in Operations. Analyst billet or Time since 
Graduation, Part I and Category type had a multiple 
correlation of .8210, indicating that technical knowledge is 
extremely dependent upon inherent ability and the way time 
is utilized after aquisition of these abilities. Since the 
reliability of this test is a high .902) and its overall 
correlation to a real life criterion is high, it follows that 
the quantitative adequacy of this test is very high. 


* Hereafter referred to as Q.P.R., with a maximum of 
3.0. 
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Within the assumptions stated, the quantitative 
reliability and validity calculation and the previous subjective 
verifications, the overall adequacy of this test is excellent. 

One final word on this "sticky" subject. This test was 
sent to Mr. R.F. Richardson, Head Operations Analyst for 
LING-TEMPCO-VAUGHT ASTRONAUTICS in Dallas, Texas 
for his comment as well as the reaction of his fellow employees. 
Mr. Richardson replied that in his opinion, and the opinion of 
his collegues, the overall design of the test should yield a 
meactical relative measure of the techmical tools cf @perations 
Analysts. 

II. Comments on the suggested method and proposed measure 
of effecticeness. 

In the suggested methodology, two families of curves were 
to be plotted and studied in hopes of arriving at a useful meas-— 
ure of effectiveness. The first family of curves was to be 
a plot of weighted adjusted score (as described in I above) for 
categories 2A, 2B, 2C versus time. The courses were post-— 


ulated to appear as in Figure ée. 


" Cc 
Q be C4 = level of knowledge for 
i a0 A eGo IAs A category | students. 
ere CLEGoRY 2C = 
7 agi “ALZEGoORY 2B= B 


Time, in months, since graduation. 
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The second family of courses were to be concerned with 
the process of learning and relearning basic technological 


knowledge and were hypothesized to appear as in Figure 3. 


WEIGHTED ADJYSTED 


aN CATEGORY 2D= D 
my 5 

~ 

e. i. a ne CATEGoRy 32 E 
ie , 

a 

a 25 


Time, in months, in Operations Analyst ee 
Pleura 

The plots of Figure 2, if correct, would indicate that 
intervening duty assignments between acquisition of technical 
knowledge and application in the given field results in a 
larger loss of the tools required. A useful M.O.B. was to 
be constructed from the type graphs of figure 3. For 
example: if "b" in figure 3 indicates the ordinal value of 
the asymptote of curve E, the ratio of curriculum time to 
curriculum time plus time required for category 2D (graduates 
who are essentially re-learning eve technical tools) would 
yield a value between O and 1, a higher value correlating to 
a more effective utilization of the technical resorurce and 


would imply a certain percentage of effectiveness. If this 
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ratio were to be drastically low it would demonstrate the 
need for assigning the technically trained to associated 
billets immediately after graduation to more effectively 
utilize the costly process of educating these individuals. 
Appendix III contains plots of weighted adjusted 
scores on Part II versus Time for the above mentioned 
categories. ‘These graphs have the approximate shapes of 
Figures 2 and 3. At this point this particular plan of 
attack breaks down. As previously stated, 34 returns 
out of a wy on oe of 197 was realized. ‘This resulted in 
a maximum of 1} points for the learning curve for category 
3 and a minimum of two points for category e<C. Itisa 
well known fact that little credence can be placed ina 
statistical procedure unless: 
I. The sample size is very large (.9 to .95) in proportion 
to the population size, SaaS 
II. One can safely assume knowledge of the exact distri- 
bution of the random variables observed, and/or have control 
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over the method of selecting the sample size. 


On Wamatons rR. Soa fPand May, Dec. Hamdboekwor 
Probability and Statistics with Tables, Handbook Publishers, 
Inc., Sandusky, Ohio 1953: 170 

"Ibid: 170-178 
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Clearly none of these proposed curves contain data 
satisfying any of these conditions. At most, one can say 
that the data so far gathered crudely support the hypothe- 
sized curves and that future study along these lines has 
more than a possibility of being fruitful. 

If, however, we consider the sample size to consist 
of only two groups, Operations Analysis graduates and Non- 
Operations Analysis graduates, several meaningful implications 
can be garnered from the test results. (Group I Operations 
Analysis Graduates, Group II Non-Operations Analysis 
Graduates). The mean score on Part I for Group II was 
6.0 points below Group I indicating a significantly lower 
proficiency in the area of problem solving. The mean scores 
on Parts II and III were respectively, 3 and) times as high 
for Group I as for Group II. These mean scores had a 
standard deviation of less than 7.0 for the "worst" case. 
Group I's comments on the courses that they thought they 
were lacking were of a highly specialized nature (Dynamic 
Programming, Specialized Methods of Cost-Effectiveness, 
etc.) while Group II's overall response indicated they were 
lacking in the most rudimentary areas such as basic prob- 
ability and statistics, linear programming, calculus, etc. 


Only two officers in Group II had any postgraduate education. 
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These disparities in relative performance of Group II to 
Group I exist despite the fact that the mean time in 

billet for Group II is 19 months = only two months less 
than time required for completion of the Operations 
Analysis Curriculum, thus indicating that on-the-job 
training is not very effective. From these facts there 
exists a clear implication that there is a significant 
difference in the technical abilities and therefore effective~ 
ness of Group I and Group II as practicing Operations 
Analysts. ‘The least one could say is that Group I displays 
a marked advantage over Group II in the technical tools 
needed in the field of Operations Analysis. 

One might say this is not surprising —- any logical 
person would safely assume an Operations Analysis trained 
person to be more effective than one not so trained. ‘This 
conjecture is most likely true in highly specialized fields 
(such as microelectronics), but cannot be safely assumed 
for fields such as Operations Analysis. Operations Analysis 
is concerned with optimal solutions to real life problems 
arrived at by the applications of all sciences and the rational 
logic intrinsic to them. Therefore it could just as well be 
said that an officer with a good working knowledge of "the 
sciences" has an excellent chance of being an effective 
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Operations Analyst. Although all examinees in Group II 
have Bachelor's degrees in fields requiring the basic math- 
ematical and scientific technologies used by the Operations 
Analyst, the quantitative results of this study indicate 
that their ability to effectively apply these tools to 
Operations Analysis problems is relatively low. 

Why is all this important? The Navy has stated that 
its policy in the future will be to continue to educate 
officers at the Postgraduate level in ever-increasing 
iiarens.” Whatever their reasons may be for arriving at 
this decision, the cost of training these officers has or 
will be expended and cannot be recovered. It therefore 
logically follows that the most effective use of these 
officers will yield the maximum return for the money spent. 
In the past, many Operations Analysis trained officers have 
never held Operations Analyst billets and many officers 
have been assigned to these billets so long after their 
training as to nullify the maximum effectiveness they could 
aise It is highly likely that this situation exists in other 


technical fields within the Navy. 


SO PNAV Instruction 1040.2 dated 9 December 1963: 2 


7Roman, FP. D., Russel, K. B., and Dunlop, J. M. 
A Suggested Method for Measuring the Effectiveness of the 
Utilization of Technically Trained Personnel, U.S. Naval 
Postgraduate School, Monterey, California 1964: 9 
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These results clearly indicate that the Navy should 
give more thought to the problem of using the technical 
abilities of these officers more effectively. Specifically, 
a review of present career planning should be made in 
hopes of arriving at a method of duty assignment planning 
which will allow technically trained officers to be assigned 
associated billets compatible with their education at the 
earliest possible date after completion of this education. 
This of course must be done in light of providing the 
officer with the operational experience necessary for 


10 The authors are certain that the 


Military Command. 
Navy is aware of this problem, but may not be cognizant 
of the fact that the use of officers not trained in such 
specialties as Operations Analysis results in a large loss 


of effectiveness that could be realized by more judicious 


use of the available corps of technically trained officers. 


10 SC ROCNAV Instruction 1520.4 dated 7 March 1963: 2 
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IIt. Prediction of performance in Operations Analysis 
Training. 


During the gathering and analysis of the data for the 
previous parts of this paper the authors discovered what 
appeared to be another fruitful avenue of investigation; 
the prediction of performance in the Operations Analysis 
Curriculum. Since Q.F.R. is accepted universally as a 
measure of a person's knowledge of the technical tools 
acquired during his training, it would be especially welcome 
if some means of predicting, before training commenced, the 
approximate skill any particular person, or more appropriately, 
any group of persons would aquire. If such a predictor 
could be developed, any input group to the Operations 
Analysis Curriculum could be selected so as to yield maximum 
benefit to the Navy for the time and money expended on the 
training. 

In an attempt to develop some prediction relationships, 
a number of statistics were considered as variables. Using 
Q.P.R. as the variable to be predicted nine other statistics 
were treated as the predictors; 

l. Score on Part I of the test vehicle. 

ek. Verbal score on the Graduate Record Examination. 


os Quantitative score on the Graduate Record Examination. 
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4h. Advanced score on the Graduate Record Examination. 


5. Time since graduation or time in an Operations 
Anslysis billet. 


6. Score Part II of the test vehicle. 

7. Score Part III of the test vehicle, 

8. Category as defined previously for responders. 

Gar Sum of Fares Ig Siig and fii, 

A standard linear regression analysis (see Appendix IV) 
was performed on a CDC 160) to obtain the prediction 
equations. ‘The level of significance on the F-test used 
was fixed at 0.01 for all runs. As previously mentioned 
in Part II of this paper one run was made using the scores 
of Part II and then Part III of the test vehicle as the 
dependent variable. Many other equations were also obtained, 
not with the purpose of predicting Q.F.R. specifically but 
for obtaining as much information about the correlations of 
the various statistics as possible. A complete summary of 
these equations is contained in Appendix IV. Initially three 
different groups of data points were used: 

I. All Operations Analysis students and graduates 
for which Q.FP.R., and the G.K.EB. scores were available. 

II. The class of 1965. 

III. The class of 1965 and previous Operations Analysis 
graduates whose statistics were available. 


Oye 





From each group of data points eight regression 
equations were obtained. All these equations are tabulated 
in Appendix IV. The equations of interest in predicting 
Q.P.R. are necessarily those containing only independent 
variables which are readily available before a person or 
person's start the course of instruction. Of the group 
of variables used in the analysis only Part I and the G.R.E. 
scores would be obtainable prior to enrolement. 

The following is the regression analysis and equations 


obtained in the order of the groups of data points. 
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It is readily seen from the above equations that the 
multiple correlation coefficients of the three groups of 
data points are remarkably constant (0.5978, 0.6142, 
0.6118). The authors found this somewhat surprising 
since previous analysis of data had indicated that the 
Q.P.R. of the 1966 class would not have settled down 
into the same pattern as those of the 1965 class and the 
graduates. It had further been hypothesized that the 
data points obtained from graduates would be somewhat 
distorted due to their high mean Q.FP.R. and the fact 
that since they were not in school they would have lost 
some of their "test taking ability." Even though the 
multiple correlation coefficients are fairly constant over 
the three groups of data it is apparent that the class of 
1966 does degrade the results and it is therefore felt that 
either the second or third equation should be used for 
prediction. 

As time passes and more data points are obtained it 
is felt that this method will yield ever better prediction 
equations. The authors leave it to the readers to decide 
if the present equations with approximately 0.3) as the 
Standard error of estimate is sufficiently accurate for 
their application. The authors do feel that the results 


a5 








obtained thus far strongly support the hypothesis that the 
predictors (Part I, G.R.E. scores) can be used to select 
the makeup of a class or predict the performance of a 
previously selected group. It is further felt that these 
predictors and methodology could be applied to other 


curricula with equally promising results. 


26 





CHAR aE rei 


CONCLUSIONS AND RECOMMENDATIONS 


Reliability and Validity determinations for the test 
instrument have shown that the basic tool of the proposed 
methodology is a feasible and meaningful model with which to 
measure technical resources. Admittedly there are a number 
of necessarily subjective "proofs" included in this analysis, 
but the authors feel that quantitative determinations have 
served to augment these "proofs," and provide a creditable 
coalescence leading to an overall high degree of adequacy for 
the test vehicle. ‘This logical trajectory had led the authors 
to the conclusion that this test does measure relative 
percentages of technical resources which in turn gives an 
excellent indication of effectiveness in billets requiring these 
resources. 

Analysis of the data garnered from this test model on 
the Operations Analysis Curriculum has objectively indicated 
thats 

(1) The percentage of technical resources retained 
decreases logarithmically (approximately) with time. 

(2) On-the-job training in Operations Analyst billets is 


significantly inferior to formal Operations Analysis training - 


cal 











especially in the area of learning fundamental technical 
knowledge. 

(3) If technical ability is utilized immediately after 
it's acquisition, loss of this resource occurs at a slower 
rate. 

(4) Although Operations Analysis is a field utilizing 
mainly the basic sciences (Mathematics, Physics, Probability, 
etc.), College Graduates (Non Operations Analysis trained) 
holding degrees which require knowledge of most of these 
Subjects do not possess nor readily learn the basic tools 
of the Operations Analyst. 

(5) Graduates of the Operations Analysis Curriculum 
(regardless of time since completion of course or use or 
non-use of technical abilities) display three (3) to four () 
times the technical knowledge of College Graduates who have 
held Operations Analyst billets for an average of 19 months. 

(6) Use of Part I of the test instrument and the three 
parts of the graduate achievement tests taken by all students 
of USNPGS has yielded a quantitative method of augmenting 
the screening of prospective students to the Operations 
Analysis Curriculum. 

From these conclusions it is recommended thats: 

(1) Further investigation of this methodology be under- 


Qe) 





taken. It is obvious that the sample sizes obtained were 
not large enough to make "bullet proof" statistical conclusions 
but rather serve as indications that the stated postulates 
concerning the use of technical abilities and the effectiveness 
of various groups of officers holding Operations Analyst 
billets are essentially correct. 

(2) The significant disparity in technical abilities of 
the two groups of officers mentioned in 5. (above) definitely 
suggests that Naval Officer assignment policies be reviewed 
in hopes of assigning as many Operations Analysis trained 
officers to Operations Analyst billets as is consistent with 
other requirements. 

(3) Use of the regression equations and correlation 
coefficients presented in Appendix IV would be of consider- 
able use for Operations Analysis Curriculum officials in 
balancing the levels of Operations Analysis Classes in any 
manner consistent with the ever changing policies and needs 
of the Postgraduate School. 

(4) If future investigations of this type are undertaken 
a more effective method of insuring the test subjects complete 
the required questionaire and test instrument should be invoked. 
The authors were forced to appeal to the examinees sense 


of responsibility, their response being completely voluntary. 
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It is felt that in order to obtain meaningful sample sizes 
from which learning and forgetting curves can be accurately 
constructed, and measures of effectiveness deduced, 


completion of the test vehicle should, in Some manner, be 


made mandatory ‘ 
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RELIABILITY CALCULAIWe hs 
Data from Table #1 


Equations from Chapter II 
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and Forgetting Curves 
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Weighted Adjusted Score is the ratio of the Mean Score 
of a particular category to the individual's score on 


Part I multiplied by the individual's score on Part II. 
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APPENDIX IV 


REGRESSION EQUATIONS 
AND 


CORRELATION COEFFICIENTS 
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GROUP I 
Regression Equations using all Operations Analysis students and 
graduates for which QPR, and the G.R.E. scores were available. 


Variable No. 

Quality Point Rating 

Part I Score 

G.R.E. Verbal Score 

G.R.E. Quantitative Score 

G.R.E. Advanced Score 

Time (in billet or out of school) 
Part II Score 

Part III Score 


MO~YWOR mE WY 


CORRELATION COEFFICIENTS 


ROW 1 
1.00000 42051 39584 o3737h 2 50869 
ROW 2 
42051 1.00000 04,7756 030214 042939 
ROW 3 
03958h -47756 1.00000 slgnoe 041359 
RW 4 
eo 3 30214 19532 1.00000 035157 
ROW 5 


50869 042939 041359 035157 1.00000 


SAMPLE SIZE 77 
NO. OF VARIABLES 5 NO. OF VARIABLES DELETED 3. (FOR VARIABLES 


DEPENDENT VARIABLE IS NO NO. 1 DELETED, SEE BELOW) 
COEFFICIENT OF DETERMINATION 0357k 

MULTIPLE CORR. COEFFICIENT -5978 

SUM OF SQUARES ATTRIBUTABLE TO REGRESSION 10259485 


SUM OF SQUARES OF DEVIATION FROM REGRESSION 19.04868 


VARIANCE OF ESTIMATE 264,56 
STD. ERROR OF ESTIMATE .51436 


INTERCEPT (A VALUE) -1.14,097 
STD. ERROR OF INTERCEPT .57577 
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GROUP ITI 
Regression Equations using just the class of 1965. 


Variable No, 

Quality Point Rating 

Part I Score 

G.R.B. Verbal Score 

G.R.F. Quantitative Score 
G.R.&. Advanced Score 

Time (in billet out of school) 
Part II Score 

Part III Score 


OIA NH WY! 


CORRELATION COEFFICIENTS 


ROW 1 
1.00000 SSabs ety A927 = 643350 2 D4L4 5 
ROW 2 


°31319 1.00000 043538 .26674 009552 


ROW 3 

A927 »43538 1.00000  .44631 eorteD 
ROW 4 

043350 «26674, 044631 1.00000 045348 
ROW 5 


2 LAL S 009552 037h25 =k 5348 = 1.00000 


SAMPLE SIZE 36 
NO. OF VARIABLES 5 NO. OF VARIABLES DELETED 3 (FOR VARIABLES 


DEPENDENT VARIABLE IS NOW NO. 1 DELETED, SEE BELOW) 
COEFFICIENT OF DETERMINATION 03772 

MULTIPLE CORR. COEFFICIENT 6142 

SUM OF SQUARES ATTRIBUTABLE TO REGRESSION Beco? 


SUM OF SQUARES OF DEVIATION FROM REGRESSION 3.66991 


VARIANCE OF ESTIMATE ~11838 
STD. ERROR OF ESTIMATE .34407 


INTERCEPT (A VALUE) = 10242 
STD. ERROR OF INTERCEPT .64,905 
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GROUP III 
Regression Equations using the class of 1965 and previous 
Operations Analysis graduates whose statistics were available. 


Variable No. 

Quality Point Rating 

Part I Score 

G.R.E. Verbal Score 

GoR.E. Quantitative Score 
G.R.E. Advanced Score 

Sum of Parts I, II, and III 
Part II Score 

Part III Score 


O~YAMNMEWND FE 


CORRELATION COEFFICIENTS 


ROW 1 
1.00000 034292 -41106 «36267 «46906 
ROW 2 
34292 1.00000 oA 7445 BAS Say, 14136 
ROW 3 
41106 47445 1.00000 41974 235087 
ROW 


«36267 «28329 41974 1.00000 sobDOe 
ROW 5 
46906 14136 - 35087 ~51502 1.00000 


SAMPLE SIZE 45 
NO. OF VARIABLES 5 NO. OF VARIABLES DELETED 3 (FOR VARIABLES 


DEPENDENT VARIABLE IS NOW NO. 1 DELETED, SEE BELOW) 
COEFFICIENT OF DETERMINATION ~37h3 

MULTIPLE CORR. COEFFICIENT .6118 

SUM OF SQUARES ATTRIBUTABLE TO REGRESSION 2.81137 


SUM OF SQUARES OF DEVIATION FROM REGRESSION 4.69891 


VARIANCE OF ESTIMATE e1l7h7 
STD. ERROR OF ESTIMATE o3h27h 


INTERCEPT (A VALUE) ~02897 
STD. ERROR OF INTERCEPT  .53596 
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GROUP IV 
Regression Equations using only returns from graduates of the 
Operations Analysis Curriculum or officers holding Operations 
Analyst billets. 


Variable No. 

Time (in billet or since graduation) 
Part I Score 

Part II Score 

Part III Score 

Category (2A=1.0, B=2, 2=3, D=4, 3=5) 


Wi FW db 


CORRELATION COBFFICIENTS 


ROW 1 
1.00000 -.08659 o 20970 012543 


ROW 2 
-.08659 1.00000 -.18069 36119 


RW 3 
-20970 + ~.18069 1.00000 =.72901 


ROW 4 
012543 036119 = =.72901 1.00000 


SAMPLE SIZE 34 
NO OF VARIABLES 4 NO. OF VARIABLES DELETED 1 (FOR VARIABLES 


DEPENDENT VARIABLE IS NOW NO. 3 DELETED, SEE BELOW) 
COEFFICIENT OF DETERMINATION  .674,0 
MULTIPLE CORR. COEFFICIENT 8210 
SUM OF SQUARES ATTRIBUTABLE TO REGRESSION 25.26.8069), 


SUM OF SQUARES OF DEVIATION FROM REGRESSION 1222.22490 


VARIANCE OF ESTIMATE 40.7083 
STD. ERROR OF ESTIMATE 6.38285 


INTERCEPT (A VALUE) 5.60460 
STD. ERROR OF INTERCEPT 9.01651 
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